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Abstract. The prevalence of human error in safety-critical occupations remains a major challenge to mission success despite 
increasing automation in control processes. Although various methods have been proposed to prevent incidences of human error, 
none of these have been developed to employ the detection and regulation of Operator Functional State (OFS), or the optimal 
condition of the operator while performing a task, in work environments due to drawbacks such as obtrusiveness and impracticality. 
A video-based system with the ability to infer an individual’s emotional state from facial feature patterning mitigates some of the 
problems associated with other methods of detecting OFS, like obtrusiveness and impracticality in integration with the mission 
environment. This paper explores the utility of facial expression recognition as a technology for inferring OFS by first expounding on 
the intricacies of OFS and the scientific background behind emotion and its relationship with an individual’s state. Then, descriptions 
of the feedback loop and the emotion protocols proposed for the facial recognition program are explained. A basic version of the 
facial expression recognition program uses Haar classifiers and OpenCV libraries to automatically locate key facial landmarks 
during a live video stream. Various methods of creating facial expression recognition software are reviewed to guide future 
extensions of the program. The paper concludes with an examination of the steps necessary in the research of emotion and 
recommendations for the creation of an automatic facial expression recognition program for use in real-time, safety- critical missions. 


1.0 Introduction 

The increasing mechanization of 
control processes in the modern world has 
lessened the gross total workload of the 
operator and has increased the breadth of 
tasks that humans can accomplish. 

However, despite the operator shouldering 
an objectively smaller workload than before 
the advent of automation, human error still 
accounts for the largest percentage of 
actions leading to the prevention of mission 
success. When affected by emotional and 
environmental influences, the operator can 
falter in the execution of critical tasks. In 
order to combat the negative consequences 
of a sub-optimal operator state, it becomes 
necessary to create a system that both 
evaluates the operator and influences him 
or her into embracing an optimal emotional 
state. Operator Functional State (OFS) 
defines the current theory of what 
constitutes an optimal state: “the 
multidimensional pattern of human psycho- 
physiological conditions that mediates 
performance in relation to physiological and 
psychological costs, [and] results from the 
synthesis of operator characteristics, current 
operator conditions, and the operator’s 
interaction with operational 
requirements. ”[13] OFS does not place 
value judgments on a specific aspect of 
state, but rather looks at its holistic 


representation and effect on the operator. 
The regulation of OFS in operators could 
help to spot and prevent any human errors 
from occurring. 

Various techniques have been 
proposed and utilized to measure OFS in 
real-time, such as electroencephalography 
(EEG), eye activity, and core temperature 
measurements. [13] The implementation of 
these techniques allows for the least noisy 
sources of data; however, each method only 
measures a single aspect that affects OFS, 
and the implementation of the multiple 
apparatuses measuring these different 
methods onto a single person would cause 
considerable discomfort and diminishing 
returns in respect to the amount of useful 
data acquired. The most important 
component of assessing OFS becomes 
finding a measurement that can take 
multiple factors into account while 
extrapolating from a single source of 
information. This source of information can 
be found in the human face. 

Humans have residual information 
about their current activities and emotions 
stored in their face for a variable length of 
time, and can be charted through tracing 
muscle movements. Researchers have 
employed electromyography (EMG) in order 
to measure muscle activity in the face 
because of its reliability in detecting facial 
muscle movement. [9] However, the method 
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uses electrodes attached to the face, which 
has several technical and practical 
drawbacks. [9] In the setting of a flight deck, 
for example, the inconvenience of attached 
electrodes overshadows the benefits. A 
potential non-obtrusive system comes in the 
form of monitoring an operator through live- 
video streams, and by using facial 
expression as a measure of state. This 
paper overviews the science behind facial 
expressions, their link to emotions, and 
describes stress, fatigue, and complacency: 
three psycho-physiological states that 
promote sub-optimal OFS. Next follows a 
description of the feedback loop video 
system, along with the suggested protocols 
and models for the regulation of an 
operator’s state in the case of stress, 
fatigue, and complacency. Then, a 
description of the basic program created by 
the author, and an explanation of the 
computer algorithms and statistical models 
available to create a facial expression 
recognition program shows the extent of the 
possibilities in computer vision. 

2.0 Emotion and OFS 

Darwin’s work, The Expressions of 
Emotion in Man and Animals, postulates 
that human emotions, and subsequently the 
physical generation of emotion through 
facial expressions, are “evolved and 
adaptive.” Darwin proposes that human 
emotion and its expression could not be the 
product of societal constructs, but represent 
the culmination of millions of years of 
evolution selected to benefit the species. [8] 
As social creatures even before the advent 
of human speech, ancestral humans could 
communicate their emotions and intentions 
to others through their facial expressions. [8] 
Despite the formation of language, humans 
keep their ability to show their emotional 
state through the face. Ekman and his 
colleagues later revived interest in Darwin’s 
theory of universality of emotion by 
continuing cross-cultural studies and noting 
the similarity of emotional expressions 
throughout the world. Ekman and his 
colleagues agreed with Darwin and proclaim 


that facial expressions, and the emotions 
displayed by each facial expression, are 
universal. [10] In order to create a scientific 
and methodological manner of determining 
a facial expression, Ekman also created the 
Facial Action Coding System (FACS), which 
uses facial muscle movements, defined as 
Action Units (AUs), to discern emotion. 
Vetted by the scientific community, FACS 
presents a foundation for all measures of 
facial expression and continues to be a 
measuring scale for human emotion. FACS 
allows the scientific examination of facial 
muscles in facial expression analysis, which 
can be used to measure OFS after 
examining the effects of emotion. 

Emotion has a continual effect on an 
operator’s perception of a task and on his or 
her OFS. Studies have shown that 
emotional arousal has an effect on the 
general reactivity and decision-making 
abilities of an individual, despite the 
emotion’s valence. [16] The impact of 
emotion on cognitive tasks elevates it to an 
aspect of state to be examined. The face 
allows a unique combination of both 
physiological and psychological changes in 
the body. For example, the perception of 
happiness in an individual connotes an 
emotional recognition fueled by the 
physiological conformation of different 
muscles in the face. The universality of 
facial expressions also validates the 
creation of a facial recognition system 
focused on the determination of sub-optimal 
OFS since it would not have to be calibrated 
for each individual operator. The different 
conformation of muscles can be measured 
through the FACS, which already decoded 
the different potential muscle movements in 
the face. Although different states of sub- 
optimal OFS exist, the following three 
incorporate a large spectrum of the 
challenges faced by operators and will be 
concentrated. 

2.1 Stress 

The Lazarus definition of stress, “the 
process of appraising [negative or 
cumbersome] events, of assessing one’s 
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potential to control or cope with the event, 
and continuing reappraisal as new 
information becomes available,” defines 
stress as a process for dealing with harmful, 
emotional events rather than a strict 
emotional state in itself. [14] Stress does not 
constitute an emotion, but rather combines 
an amalgam of emotions in reaction to a 
situation. Stress can arise from a time- 
restricted and work-intensive environment, 
like those found in safety-critical missions, 
and has both psychological and 
physiological effects on the body that leads 
to a sub-optimal OFS. 

In terms of the psychological effects 
of stress, the brain attempts to mount 
multiple defensive measures, such as a 
reflexive completion of tasks without 
cognitive input, in order to reduce 
stress. [14] However, following the failure or 
lack of initialization of these defensive 
measures, the mind suffers from a reduced 
ability to process the environment due to the 
high workload. 

Subsequently, stress affects an 
individual’s physical state as well. Selye’s 
General Adaptation Syndrome (GAS) 
represents the process in which stress 
manifests in the physiology of an organism. 
GAS defines the “progressive responses to 
prolonged stress in which an organism 
mobilizes for action and compensates for 
stress,” and its component, Alarm, 
Resistance, and Exhaustion (ARE), shown 
in Fig. 1 , delineates the stages of the 
response. [14] The GAS shows the general 
wearing down of an organism in the face of 
stress. Ultimately, when stress fails to be 
mitigated, the organism reaches exhaustion 
and becomes unable to respond 



Figure 1. Alarm, Resistance, Exhaustion 
(ARE).[14] Figure shows the three stages 
of Selye’s General Adaptation Syndrome 
(GAS). 


appropriately to the environment. The 
culmination of stress, psychologically and 
physiologically, creates an error-prone 
environment for the operator due to his or 
her inability to cope with high-load tasks. 

2.2 Fatigue 

Fatigue colloquially describes 
“feelings of fatigue, exhaustion, and to 
problems of attention and motivation.”[5] 
However, the amorphous boundaries of this 
definition of fatigue encompass a multitude 
of other emotional states like drowsiness 
and intense hunger. In the context of work- 
related fatigue, fatigue defines a “normal, 
healthy response to an oncoming depletion 
of resources, caused by the execution of 
physical and mental tasks,” which highlights 
the fact that humans are not meant to 
concentrate on a task for an extended 
period time without regard for their overall 
state of being. [5] Work-related fatigue does 
not include the effects of outside influence 
like lack of sleep, although these factors can 
be confounded and mistaken as fatigue. 
Fatigue occurs in a work environment due 
to extended interaction with a task without 
any breaks or change of requirements in the 
task. Fatigue affects the operator’s psycho- 
physiological state by inducing a state of 
tiredness usually followed by a decrease in 
cognitive function. 

The effects of fatigue leads to the 
operator being unable to completely 
process his or her environment. The 
monotony of the task causes the operator to 
revert to more energy-saving strategies. For 
example, in response to the decrease in 
awareness, the operator either continues 
the task at his or her current pace and 
makes mistakes, or slows down the process 
in an attempt to reduce the mistakes. [5] 
Essentially, the operator has to take the 
time to evaluate different work strategies to 
combat the side effects of fatigue. 

2.3 Complacency 

Complacency, in the terms of safety- 
critical missions, defines the tendency for 
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operators to blindly trust automation without 
monitoring, and often renders the operator 
unable to quickly respond to any 
malfunctions. [12] Complacency also results 
when the operator works in a highly reliable 
and automated environment, and the 
operator’s sole task is to stay vigilant in 
case of occasional error. [1 5] Although 
automation has increased the capability of 
an operator in a multitude of environments, 
it has also caused an unbalanced mental 
workload and reduced situational 
awareness in operators. [ 1 2] [ 1 5] 
Complacency differs from boredom due to 
the implicit trust in the automation; 
complacency lulls the operator into a false 
sense of security due to trust. While an 
operator may be bored because of the 
absence of a task, a complacent operator 
can also be bored due to the trust given to 
the automation. A complacent operator 
appropriates a small cognitive investment in 
his or her environment, and can lead to 
overlooking any number of mistakes in the 
automation. 

Complacency does not necessary 
increase the number of errors committed by 
the operator, but rather causes the operator 
to ignore potentially dangerous situations 
due to a lack of cognitive awareness. 
Complacency leads to a reduced reaction 
time if any errors occur, or an inability to 
spot any errors at all. If an error occurs 
while the operator is complacent, the 
operator could be sent into a panic while 
attempting to discover and fix the problem 
occurring in the task simultaneously. 

3.0 Video Systems and Facial 
Expression Recognition 

3.1 The Feedback Loop 

The use of a feedback loop allows 
information gathered from the user to be 
analyzed in real-time and sent back to 
inform him or her to modify or continue his 
or her current actions. The proposed 
feedback loop, as shown in Fig. 2, notifies 
the user about their state based on the data 
analyzed from the facial expression 


Figure 2. The Feedback Loop. The feedback 
loop utilized in the potential facial recognition 
system in order to aid pilots and other critical- 
safety task workers. 

recognition program. The program then 
sends the product of the data analysis back 
to the user in the form of useful information 
regarding their current state. For example, 
when the subject displays a sub-optimal 
state of awareness, the feedback loops 
allows this information to be sent back to 
the user. Users can then modify their state 
based on their newly acquired knowledge, 
or, if they lack the ability due to extreme 
conditions, allow other protocols to be put 
into place. The feedback sent to the 
operator would depend on the emotion he 
or she displayed, and would be designed to 
combat that specific sub-optimal state. The 
different protocols that would be assigned to 
the three main sub-optimal OFS, and the 
make-up of the physical systems in the 
feedback loop, are explained further. 

3.2 Suggested Protocols for Sub- 
Optimal States 

3.2.1 Stress Protocol Description 

The weight of stress on an operator 
leads to a variety of negative emotional 
responses such as fear, anger, disgust, 
distress, and helplessness. [4][1 1] Although 
a distinct stress progression does not 
project itself directly onto the face, the 
stress can be traced through the shifts 
between negative emotions on the face in 
response to high-load work. The program 
would monitor the operator for any 
sequence of negative facial expressions 
over a period of time. Current research 
shows that the detection of stress through 
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live-video streams is both possible and 
improving and could be an integral part of 
assessing OFS.[4] 

The program would constantly 
monitor for an emotional reaction to 
stressors, like a sudden change in the 
environment, and note the duration of the 
emotional response. If stress episodes are 
infrequent and short in duration, the 
program will not report those results to the 
operator. However, once the length or 
frequency of the episodes increase, then 
the program would report the findings to the 
operator through voice-command or through 
a notice in one of the display monitors. The 
program then prompts the operator to 
commence a relaxation technique, such as 
deep breathing or voluntary and directed 
muscle contraction. The implementation of 
the stress protocol hopes to reduce stress 
as it begins, rather then allowing the 
operator to reach maximum stress levels at 
any point. However, if the levels of stress 
are deemed to be overwhelming the 
operator, then the program will suggest 
replacing the operator if possible. 

3.2.2 Fatigue Protocol Description 

Although the measurement of 
fatigue should not include the symptoms of 
sleep loss or an out-of-sync circadian 
rhythm, the outward manifestation of these 
states are similar.[5] Therefore, for the 
purposes of the fatigue protocol, all states 
that manifest similarly to fatigue will be 
considered as such. However, while fatigue 
can be alleviated through the 
implementation of short breaks or modifying 
the task, the curing of symptoms caused by 
sleep loss require more intensive recovery. 
In light of the similarities between the states, 
the program will also pay attention to the 
duration that the emotion outwardly affects 
the operator. 

Although the current literature does 
not describe the facial expressions 
associated with fatigue, the proposed 
program looks for characteristics like heavy- 
lidded eyes combined with a neutral 
expression and possibly the inability to 


detect the operator’s face for increments of 
time due to head-nodding or complete eye 
closure. The program monitors the operator 
for their level of fatigue throughout the task. 
The goal of the program is to squash any 
signs of fatigue before they affect the 
operator in a fundamental way and reduces 
work efficiency. If the level of fatigue passes 
a level to be determined, the program would 
inform the operator and instruct them to 
take a break, focus on another task, or 
instruct the task-manager to appropriate 
another task to the operator. 

3.2.3 Complacency Protocol 
Description 

Complacency generally manifests 
itself in ways similar to boredom. In terms of 
OFS, complacency usually occurs from the 
under-load experienced due to automation. 
Research shows that the most effective way 
to place the operator back into experiencing 
the appropriate level of load comes in the 
form of adaptive automation. [12] Adaptive 
automation refers to the ability to reallocate 
the level of automation from the operator to 
the environment depending on the 
operator’s state. [12] 

The current literature does not 
describe the facial expressions associated 
with complacency, however, for the 
purposes of the program, complacency will 
be discerned through the formation of a 
neutral face, combined with the appearance 
of cues like constant pupil movement away 
from potential locations of display monitors. 
If the operator appears complacent, then 
the program will alert the adaptive 
automation system to return some function 
back to the operator. The transfer of work 
back to the operator will allow them to 
regain focus and invest some cognitive 
awareness into his or her environment. 

3.3 The Physical Components 
3.3.1 Webcam 

Since their introduction, webcams 
have improved in both their popularity and 
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their capabilities; however, they still suffer 
from a multitude of technological 
shortcomings, such as the maximum 
number of frames per second and 
megapixels. The webcam also experiences 
some practical limitations, like the inability to 
function well in poorly lit environments. [9] 
However, despite these setbacks, webcams 
allow an inexpensive alternative to 
expensive video equipment; webcams are 
built to stream video live onto another 
source, while more traditional video 
equipment solely stores video for later use. 
Webcams also have the advantage of size, 
and the introduction of a webcam into the 
work environment will be almost 
unnoticeable due to their small size. 

3.3.2 Thermal Camera 

Preliminary research shows that 
thermal cameras present another non- 
obtrusive way to detect facial muscle 
movement. 9 By analyzing the muscles 
delineated in the FACS, the researchers 
found that the production of AUs creates an 
increase in the temperature in certain areas 
of the face while decreasing other areas, 
and shows a promising method of 
evaluating facial muscle movement. [9] 
Programs that rely on webcams suffer from 
the camera’s inability to register images 
correctly when the subjects are under poor 
lighting. Thermal cameras, however, do not 
require any light at all; the images form 
through the analysis of heat emanating from 
a surface. A drawback of thermal cameras 
comes from the camera’s inability to see 
through other objects. For example, if an 
operator had to wear goggles, the thermal 
camera would only reflect the heat of the 
goggles rather than any of the eye muscles 
due to the goggles occluding the eyes. The 
occlusion occurs for all objects, regardless 
of transparency, so any accessories that 
cover the face will have to either be 
removed or redesigned to prevent facial 
occlusion. 

The combination of both standard 
webcams and thermal cameras in the 
system would allow for both the continual 


monitoring of the operator despite lighting 
conditions and the ability to corroborate 
facial muscle movements between the 
webcam and the thermal camera when the 
operator sits under good lighting. The 
utilization of both methods would help to 
cover some of the drawbacks of either 
method and create a more balanced 
system. 

4.0 Computer Algorithms for 
Deciphering Facial Expressions 

The current iteration of the automatic 
facial recognition program, created in Visual 
C++, utilizes Open Computer Vision 
(OpenCV) libraries and Haar cascades, and 
tracks a human face in a live video-feed. 
After identifying a human face, the program 
then subsequently identifies some key facial 
landmarks: the eyes, pupils, nose, mouth, 
and the corners of the mouth. Some of the 
limitations of the program, both intentional 
and unintentional, are as follows: the 
program currently only tracks one individual 
at a time, has a slight delay due to the 
application of the facial landmark 
identification boxes onto a live-stream 
video, does not include thermal camera 
integration, and can only identify facial 
features deviating 30 degrees either to the 
left or right from facing the camera directly. 

The program also lacks a key 
component: facial expression recognition. 
The current state of computer vision allows 
for the utilization of a multitude of algorithms 
to create a program that automatically 
recognizes facial expression. The most 
prominent algorithms seen in recent years 
are Haar cascades, Active Shape Models, 
Active Appearance Models, and the 
Piecewise Bezier Volume Deformation 
Model. 

4.1 Haar Cascades 

Haar cascades, described by Viola 
and Jones [2001], describe a method to 
quickly identify images using feature- 
detection. Haar cascades rely on Haar-like 
features that detect the change in contrast 
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values between adjacent rectangular groups 
of pixels, rather than the intensity values of 
a pixel itself. [19] Although the use of Haar- 
like features reduces computing time 
drastically, comparing every feature in an 
image in order to find an object creates 
unnecessary work. [19] The cascading of 
classifiers allows the further reduction of 
necessary computing power. Cascading 
refers to the utilization of a degenerate 
decision tree that processes images through 
a series of comparison tests, and rejecting 
the image if it fails any one of the tests. [1 8] 
For example, if the part of the image being 
compared matches the feature in the first 
sub-window but does not match the second, 
then the cascade rejects that portion of the 
image and moves to the next feature. The 
degenerate decision tree allows for the 
comparison of all pixels without an 
unnecessarily complete classifier 
comparison, and results in the rejection of 
most negative images and the approval of 
most positive images. 

In order for a Haar classifier cascade 
to identify a particular object, the algorithm 
has to be trained. The training of the 
algorithm requires a large library of both 
positive and negative images. The positive 
images contain the object that the user 
wants the algorithm to detect, while the 
negative images consist of random images 
without the object.[18] The robustness of 
the algorithm has a positive relationship with 
the number of images used for training; 
however, this positive relationship occurs 
due to the larger probability of variety in 
larger samples. In the case of facial feature 
detection, the sample images would have to 
span across ages, genders, and races. 

For facial expression detection, 
however, Haar cascades create some 
difficulty. Facial expressions, while 
universal, can also be repressed or faked to 
some degree. While noses and eyes vary 
solely in shape and size, facial expressions 
vary in the totality of their expressions, from 
the arch of the brow to the movement of the 
mouth, and the differences between two 
different emotions could be too subtle for 
the Haar cascade to differentiate between. 


Instead of whole-face facial expression 
recognition, it could be possible to track 
more emotive landmarks of the face, like the 
eyebrows, and then create a program that 
would note the locations of facial features in 
relation to the proportions of the face, and 
then process that information for emotional 
detection. However, that process would be 
tedious and probably inaccurate for the 
subtleties of emotional expression. 

4.2 Active Shape Models (ASM) 

Cootes and Taylor's [1995] ASMs 
illustrate a trained statistical modeling 
algorithm that utilizes Point Distribution 
Models (PDMs) to rapidly locate the desired 
structure. [3] PDMs describe a training set of 
images dotted with landmark points. The 
training sets have multiple instances of the 
same object, but with some natural 
variation. The mean position of the 
landmark points in relation to the object, and 
then the points that show how each object 
in the training set deviates from the 
calculated mean, are determined. [3] After 
the establishment of the PDMs, the ASM 
performs an iterative search to locate the 
shapes of the structures in different 
images. [3] The ASMs result in the ability to 
fit the model to the image after a finite 
number of iterations. 

The human face, although variable, 
generally follows a model format with the 
same number of facial features. The general 
shapes of the facial features are also the 
same, although the size of these features in 
relation to each other, and in general, can 
change. ASMs then provide a non-rigid 
modeling approach to identify and conform 
to different facial expressions. The 
measured facial expressions would be 
trained by different sets of PDMs and then 
the conformation of certain ASMs would 
denote that expression. 

However, the use of ASMs in order 
to describe human emotion runs into similar 
problems as Haars cascades - although 
ASMs require less training, they only 
identify models defined in their training sets. 
Therefore, a PDM would have to be created 
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for each emotion. Even though ASMs are 
more forgiving than Haars cascades when 
conforming to images, the complexities of 
an emotion would have too much deviation 
from the mean. For example, the 
manifestation of a smile can range from the 
open mouth smile to the Duchenne smile. 
Each variation of the smile still constitutes a 
smile; however, the difference between a 
closed-mouth smile and an open-mouth 
smile, combined with the appearance of 
teeth, only constitute a portion of the 
variation. 

4.3 Active Appearance Models (AAM) 

AAM, also created by Cootes and 
Taylor [2001], shows a statistical modeling 
method similar to ASM, but diverges from 
tracking a specific object and moves toward 
matching a class of deformable objects. [2] 
AAM works similarly to ASM by requiring a 
training set marked with points. An image in 
the training set, after being marked, has its 
points aligned; the subsequent wrapping of 
the images around the mean of the training 
set then creates a “shape-free patch,” which 
lacks any texture. [2] The process is 
repeated for each image in the set, training 
the model to conform to a variety of shapes 
of a similar class. The “shape-free patch” is 
raster scanned into a texture vector and 
then applied with Eigen-analysis to build a 
texture model. [2] Lastly, the correlations 
between shape and texture generate a 
combination of the shape and the texture to 
create a combined appearance model. [2] 

The ability to follow deformable 
objects, such as a face with any expression, 
bestows the ability to identify the expression 
without specific training, allowing AAMs to 
surpass AAMs in facial expression 
recognition. AAM works slightly slower than 
ASM, but the added robustness and the 
reduction in different training sets for 
different emotions overrules any 
advantages of ASM in terms of identifying 
facial expression. [2] AAM, however, does 
require a large set of images for training, 
and the availability of large facial databases 
are limited. [17] Until more high-quality 


databases are established, AAMs will not be 
able to attain the robustness capable of the 
model. 

4.4 Piecewise Bezier Volume 
Deformation Model (PBVDM) 

The PBVDM utilizes the Connected 
Linear Deformation Method (CDLM), from 
the work of Tao and Huang [2003] and 
defines a model consisting of multiple 
deformable 2D patches whose movements 
are determined by a set of different vibration 
modes. [7] Vibration modes refer to the 
different displacement vectors that 
determine the deformation of a particular 
patch. These patches are connected 
together with hinges that work to maintain 
the spatial relationships between the 
patches and prevent any drifting. CDLM can 
therefore be used to track and model the 
movements of facial muscles, which would 
be represented as different vibrations of the 
patches. [7] 

The PVDM creates Bezier volumes 
across specified areas of the face, with a 
section of the surface model, or facial mesh, 
in between the top and bottom volume 
layers. The movement of the Bezier 
volumes deforms the surface model, and in 
the case of facial mesh, shows the 
movement of facial muscles in that region. 
PBVDM can utilize the AUs of the FACS to 
define the available facial movements. 
Therefore, the PBVDM allows a more 
specific analysis of desired facial muscle 
movements in real-time. [6] 

The use of PVDM in facial 
expression recognition allows a more 
scientific approach. The other models 
require training sets to tell the model what 
constitutes an emotion, and then the models 
can only identify those trained emotions. 
PVDM, on the other hand, allows an 
analysis of muscle movements through the 
use of individual volume patches without 
training sets. PBVDM can be made to be 
analogous with a FACS analysis of a face. 
Current PBVDM facial meshes do not cover 
the complexity of facial muscle movement 
covered by FACS, or the ability to decipher 
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emotions as effectively as a human can, but 
the accuracy of the algorithms is improving. 

5.0 Discussion 

Currently, the research surrounding 
the emotional states of boredom and fatigue 
are limited in both their breadth and depth. 
More research into the states and facial 
expressions associated with fatigue and 
complacency would be helpful in creating 
algorithms to define those emotional states. 
A potential method of determining the facial 
expressions associated with fatigue and 
complacency would be to conduct a study 
where individuals would be exposed to 
situations that lead to those emotional 
states, and then facial expression would be 
analyzed to search for commonalities, 
similar to the current studies on stress. [4] 

The current iteration of the program 
developed by the author allows for the 
detection of some facial landmarks, which 
represents the first step in automatic facial 
expression recognition. The program could 
be extended even further to recognize other 
landmarks like the eyebrows. In the future, 
the actual creation of an automatic facial 
expression recognition program for use in 
human factor’s experiments will help in the 
tracking of sub-optimal states of awareness 
in operators. The facial expression 
recognition program should be created 
through the use of either AAMs or PBVDM 
due to their robustness and ability to reliably 
deform to the human face. Unless more 
facial databases come into fruition, the use 
of PBVDM would be optimal due to its ability 
to deform to the human face without the 
need for high quality training sets. 

After the creation of a proper facial 
expression recognition program, a complete 
study to evaluate the emotion protocols and 
their effect on the frequency of errors during 
flight would help to determine the efficacy of 
the system. 


6.0 Conclusion 

The study of OFS requires the 
consideration of a large number of variables 
in order to find the optimal OFS. However, 
the use of automatic facial expression 
recognition could help to reduce the number 
of accidents due to human error by helping 
to reduce sub-optimal states of OFS before 
they overtake the operator. The use of video 
systems in lieu of more obtrusive methods 
of determining state allows the practical 
implementation of state-measuring devices 
into the operator’s environment. The 
implementation of video-systems in a work 
environment would help to increase the 
safety experienced by both operators and 
their charges, and lead to a safer world. 
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